Systems and methods for product visualization using a single-page application

ABSTRACT

Systems and methods are disclosed for visualizing products in a single-page application. Consistent with disclosed embodiments, an application system can receive from a client system, during rendering of a webpage received by the client system from a host system, a request for a script associated with the host system. The script can be executable by the client system to perform operations including modifying the webpage to include a visualization application in response to a selection of a visualization control in the webpage by a user of the client system. Consistent with disclosed embodiments, the application system can receive, from the visualization application, a request to display a product in a first image. A location can be identified in the first image using a machine learning model. Instructions can be provided to the visualization application for displaying the product at the location.

RELATED APPLICATIONS

This application claims priority to and is a continuation-in-part ofU.S. patent application Ser. No. 16/947,817, filed Aug. 19, 2020, nowpending, which is a continuation of U.S. patent application Ser. No.16/777,697, filed Jan. 30, 2020, now U.S. Pat. No. 10,817,648. Thisapplication claims priority to U.S. patent application Ser. No.63/199,096, filed Dec. 7, 2020, now pending. Each of these applicationsis titled “Systems and Methods for Product Visualization Using aSingle-Page Application.” Each of these applications is incorporatedherein by reference in its entirety.

BACKGROUND

A webpage may display an image of a product. But a user of the webpagemay wish to see how the product would appear in a particular location,such as her home. Adding such visualization functionality to the webpagemay be a difficult task. Relying on an external visualizationapplication (e.g., by providing a parameterized link in the webpage tothe external visualization application) may be problematic for both theuser and the operator of the website. When using such a parameterizedlink, bots and crawlers may be able to access the external visualizationapplication through the link, corrupting usage data of interest to thewebsite operator. Furthermore, the operator must design the website tocorrectly handle communication with the external visualizationapplication. Incorrect communication can result in errors affecting theuser experience, such as visualizations of incorrect products orincorrectly localized interfaces. In addition, the operator may not beable to track user interactions with the external visualizationapplication, such as what users viewed, how often they viewed it, andhow long they viewed it for. Such information may be of considerableimportance to the operator of the website.

SUMMARY

The disclosed systems and methods can enable a webpage to provideinteractive visualization of products in images selected or provided byusers. More specifically, the disclosed systems and methods can enable awebpage to provide interactive visualization of an image of a product inimages selected or provided by a user. In some embodiments, the webpagecan provide (or be modified to provide) single-page applicationfunctionality, allowing the user to select products for whichvisualizations are available and interact with the visualizations in theselected or provided images, without refreshing the webpage, opening anew webpage, or navigating away from the webpage.

The disclosed embodiments include a method for visualizing products, andmore specifically images of products, in a single-page application. Themethod can include receiving a request for a script associated with ahost system. The request can be received by an application system from aclient system during rendering of a webpage received by the clientsystem from the host system. The script can be executable by the clientsystem 110 to perform operations. The operations can include modifyingthe webpage to include a visualization application in response to aselection of a visualization control in the webpage by a user of theclient system. The method can further include receiving, by theapplication system from the visualization application, a request todisplay a product in a first image. The method can further includeidentifying a location in the first image using a machine learningmodel. The method can further include providing instructions to thevisualization application for displaying the product at the location.

The disclosed embodiments include a system for visualizing products in asingle-page application. The system can include at least one processorand at least one memory containing first instructions. When executed bythe at least one processor, the first instructions can cause the systemto perform operations. The operations can include receiving a webpagefrom a host system. The operations can further include modifying thewebpage to include a visualization application in response to aselection of a visualization control in the webpage by a user of thesystem. The operations can also include providing, to an applicationsystem in response to user interactions with the visualizationapplication, a request to display a product in a first image. Theoperations can additionally include receiving, from the applicationsystem, second instructions for displaying the product.

The disclosed embodiments include a system for enabling visualization ofproducts in a single-page application. The system can include at leastone processor, and at least one memory containing first instructions.When executed by the at least one processor, the first instructions cancause the system to perform operations. The operations can includereceiving a request to display an image of a product in a first image,from a client system in response to a user interaction with avisualization application in a webpage running on the client system. Theoperations can also include providing second instructions to the clientsystem. The second instructions being suitable for displaying the imageof the product in the first image. The operations may further includeproviding the second instructions for displaying the image of theproduct in the first image to the client system.

In some embodiments, the operations may include identifying a locationin the first image, and providing instructions to the visualizationapplication for displaying the image of the product at the location. Theoperations may comprise performing semantic segmentation of the firstimage to identify features in the first image, and selecting thelocation from at least one of the identified features. The location maybe identified using machine learning.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments or the scope of theinventions as claimed. The concepts in this application may be employedin other embodiments without departing from the scope of the inventions.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are not necessarily to scale or exhaustive. Instead,emphasis is generally placed upon illustrating the principles of theinventions described herein. The accompanying drawings, which areincorporated in and constitute a part of this specification, illustrateseveral embodiments consistent with the disclosure and together with thedescription, serve to explain the principles of the disclosure. In thedrawings:

FIG. 1 depicts an exemplary system for providing interactivevisualizations using a single-page application, in accordance withdisclosed embodiments.

FIG. 2 depicts a method for initializing a visualization application, inaccordance with disclosed embodiments.

FIG. 3 depicts a method for interacting with a visualizationapplication, in accordance with disclosed embodiments.

FIG. 4A-L depicts exemplary view of interactions with a visualizationapplication, in accordance with disclosed embodiments.

FIG. 5 depicts a schematic of exemplary computing system for performingthe envisioned systems and methods, in accordance with disclosedembodiments.

FIG. 6 depicts an exemplary method of visualization of products in adisplay interface, in accordance with disclosed embodiments.

FIG. 7 depicts a method for interacting with a visualizationapplication, in accordance with disclosed embodiments.

FIG. 8 depicts an exemplary view of interactions with a visualizationapplication, in accordance with disclosed embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to exemplary embodiments, discussedwith regards to the accompanying drawings. In some instances, the samereference numbers will be used throughout the drawings and the followingdescription to refer to the same or like parts. Unless otherwisedefined, technical or scientific terms have the meaning commonlyunderstood by one of ordinary skill in the art. The disclosedembodiments are described in sufficient detail to enable those skilledin the art to practice the disclosed embodiments. It is to be understoodthat other embodiments may be utilized and that changes may be madewithout departing from the scope of the disclosed embodiments. Thus, thematerials, methods, and examples are illustrative only and are notintended to be necessarily limiting. It is to be understood in theensuing description, that reference to providing visualization of aproduct, refers to providing visualization of an image of the product.The image may be a two-dimensional or a three-dimensional image.

The disclosed embodiments can enable a webpage to provide interactivevisualization of products in images selected or provided by users. Theproducts can include physical goods. For example, the products caninclude furniture (e.g., a table, chair, or the like), decorativeobjects or decorations (e.g., a vase, picture, or the like), functionalobjects (e.g., a lamp, consumer electronic device, or the like),construction or finishing materials (e.g., flooring, paneling ormolding, a countertop material, paint, wallpaper, or the like), or likephysical goods. The interactive visualizations can benefit users byaffording them the opportunity to view a product in a particularlocation before obtaining the product. The disclosed embodiments can bestructured as single-page applications, with the webpage modifiedas-needed to support interactive visualization, without reloading thewebpage or navigating to a new webpage. Such a single-page applicationframework can improve the user experience, as reloading the webpage ornavigating to a new webpage can disrupt the user experience, and mayalso reduce bandwidth.

A visualization application can be integrated into the webpage toprovide the interactive visualizations. Integration of the visualizationapplication into the webpage can be handled using a script, which can becustomized for the webpage. The script can configure the web browserdisplaying the webpage to perform a variety of useful functions. Suchfunctions can include determining whether the browser can support thevisualization application; determining whether products described orreferenced in the product page are available for visualization;determining an appropriate language for providing the visualization;filtering out bots and crawlers that might otherwise pollute usagestatistics; placing first-party cookies that can be used to track usersessions, as well as conversions (e.g., purchases or the like); andintegrating the visualization application with an ordering application,so that users can initiate orders from within the visualizationapplication. The script can be embedded into the webpage using a tag.Thus the functionality provided by the disclosed embodiments can beadded to a webpage without requiring extensive modifications to thewebpage. Furthermore, the visualization application may runindependently of the webpage, such that user manipulation of an imagedisplayed in the visualization application may occur without requiringreloading of the webpage.

FIG. 1 depicts an exemplary system 100 for providing interactivevisualizations using a single-page application, in accordance with thedisclosed embodiments. As depicted in FIG. 1, system 100 can includeclient system 110, host system 120, and application system 130. Clientsystem 110, host system 120, and application system 130 can beconfigured to communicate using network 140. Client system 110 can beconfigured to retrieve the webpage from host system 120 and retrieve thescript and visualization application from application system 130. Thescript can be used to integrate the webpage and the visualizationapplication, removing from the developer of the webpage the burden ofintegrating the webpage and visualization application. The visualizationapplication can allow the user to modify an image to display products,and more specifically images of products, depicted or referenced in thewebpage.

Client system 110 can be configured, according to a script, to display awebpage including a visualization of a product in an image. Clientsystem 110 can be or include an interactive computing device with adisplay. For example client system 110 can be or include a desktop, alaptop, smart phone, a tablet, or a wearable device. Client system 110can be configured to obtain the webpage from host system 120. Forexample, client system 110 can be configured with a web browser and auser of client system 110 can interact with the web browser (e.g., byentering a URL into an address bar or selecting a reference to thewebpage in another webpage, or the like) to cause client system 110 torequest a webpage from host system 120. Client system 110 can use thewebpage to implement a single-page application, in which the webpage isrepeatedly modified (e.g., in response to user interactions or data orinstructions received from host system 120 or application system 130).The webpage can include instructions that, when processed by clientsystem 110, cause client system 110 to obtain a script from applicationsystem 130. Client system 110 can be configured to execute the script.

Client system 110 can be configured, according to the script, todetermine whether the webpage can be modified to display an imagemodified to depict a product. In some embodiments, client system 110 candetermine the products displayed or referenced in the webpage. Invarious embodiments, client system 110 can determine whether applicationsystem 130 can obtain visualization information enabling modification ofan image to display a product depicted or referenced in the webpage.When such visualization information is available, client system 110 canbe configured to modify the webpage to display a control.

Client system 110 can be configured, according to the script and inresponse to selection of the control, to modify the webpage to display avisualization application. Client system 110 can use the visualizationapplication, in accordance with the script, to display the modifiedimage. The visualization application can enable a user of client system110 to select the image (e.g., by uploading the image or selecting asaved or predetermined image). Client system 110 can transmit a requestto display the modified image to application system 130. Client system110 can receive instructions for displaying the product in the imagefrom application system 130. In some embodiments, the instructions caninclude a version of the image, modified to display the product in theimage. In various embodiments, the instructions can include a model ofthe product and instructions or data enabling the visualizationapplication to display the model of the product in the image. Forexample, the instructions and data can specify at least one of ascaling, perspective, or lighting of the product in the image. Clientsystem 110 can then display the image in accordance with theinstructions received from application system 130. For example, when theinstructions include a model of the product and information describingthe scale and perspective of the object at a location in the image,client system 110 can be configured to place an image of the product,based on the model, scale, and perspective information, at the locationin the image. In some embodiments, client system 110 can be configuredto subsequently update the image based on user interaction. For example,a user can interact with the visualization application to change theposition or orientation of the product within the image. The client canthen recalculate how the model of the product is displayed, based on thechanged position or orientation information and the instructionspreviously received from application system 130.

In some embodiments, the script can be configured to manage interactionsor control communications between the visualization application and thewebpage. For example, the visualization application can create events inresponse to user interactions. These events can be handled by thescript. For example, the user can interact with the visualizationapplication to indicate an intention to purchase a product. In response,the visualization application can generate an event, which can behandled by the script. The script can configure client system 110, inresponse to the event, to initiate a purchase or add the product to ashopping cart through the webpage. Similarly the script can configureclient system 110 to set cookies in response to events generated by thevisualization application. These cookies can then be read by host system120.

Host system 120 can be configured to provide the webpage to clientsystem 110. Host system 120 can be or include a computing system, suchas a desktop, workstation, server, server cluster, cloud computingenvironment (e.g., Amazon Web Services™ (AWS), Microsoft Azure™, IBMCloud™, Google Cloud Platform™, Cisco Metapod™, Joyent™, vmWare™, or thelike), or other suitable computing system. Host system 120 can beconfigured as a web server. For example, host system 120 can provide oneor more webpages to web browsers in response to requests (e.g., HTTPrequests or the like). The webpages can be configured to providesingle-page application functionality. For example, the webpage caninclude data and instructions enabling a web browser of a client tomodify the webpage without reloading a new webpage from host system 120.In some embodiments, host system 120 can be associated with a providerof the products displayed or referenced by the webpage. For example, thehost system can provide a catalog of the products available for purchasefrom the provider (e.g., host system 120 can provide webpages for aconsumer goods store such as Home Depot, Walmart, or the like).

Application system 130 can be configured to enable client system 110 tomodify an image to include a product. Application system 130 can be orinclude a computing system, such as a desktop, workstation, server,server cluster, cloud computing environment (e.g., Amazon Web Services™(AWS), Microsoft Azure™, IBM Cloud™, Google Cloud Platform™, CiscoMetapod™, Joyent™, vmWare™, or the like), or other suitable computingsystem.

Application system 130 can be configured to provide a script to clientsystem 110. Application system 130 can provide the script in response toa request transmitted by client system 110. As shown in the followingexample, the webpage can include an element that causes client system110 to request the script from application system 130:

<script type=“text/javascript” src=“https://host address/scriptname”async></script>

In this example, the script tag can be included within the webpage(e.g., within a <head> tag or <body> of the webpage). The host addresscan be a URL pointing to a location hosted by application system 130(e.g., a directory), while scriptname can be a name of the script storedby application system 130 at that location. In this example, thescriptname can be or include a codename associated with host system 120or a product provider associated with host system 120. In this example,the location hosted by application system 130 can include multiplediffering scripts having different script names. When client system 110processes the webpage, the script tag can cause client system 110 torequest the script scriptname located at host address from applicationsystem 130. Client system 110 can then execute the script to providefunctionality as described herein.

Application system 130 can be configured to communicate with clientsystem 110 after provision of the script. In some embodiments,application system 130 can receive, from client system 110, anindication of one or more products displayed or referenced by thewebpage. Application system 130 can determine whether informationenabling display of each of the one or more products is available. Whensuch information is available for a product, application system 130 canprovide an indication of such availability to client system 110.

In various embodiments, application system 130 can be configured togenerate instructions regarding the display of a modified imageincluding the product. These instructions can be generated in responseto a request to display the modified image including the product. Therequest can include the image in which the product is to be displayed ora selection of an image (e.g., a previously stored or predeterminedimage). When an image is received (e.g., an image uploaded by the user),application system 130 can be configured to process the image todetermine image characteristics such as a perspective of the image, toidentify surfaces in the image (e.g., floors, walls, countertops,tables, or the like), to determine distances to surfaces in the image,or to obtain similar information. As described herein, such processingcan be performed using a machine learning model (e.g. neural networkmodel, or the like). When a previously stored or predetermined image isselected, application system 130 can be configured to obtain the image(e.g., receive the image from another system, retrieve the image from adatabase, computer memory, or the like, accessible to application system130, or the like).

In some embodiments, application system 130 can be configured toprovide, to client system 110, the generated instructions. In someembodiments, the instructions can include the image, or a modifiedversion of the image. For example, the instructions can include theimage modified to display the product. Additionally or alternatively,the instructions can include a model of the product (e.g., athree-dimensional model, or the like). When client system 110 does nothave the image, for example when the request includes a selection of astored or predetermined image, application system 130 can be configuredto provide the image, in addition to the model of the product.

In some embodiments, application system 130 can be configured toprovide, to client system 110, updated instructions in response to userinteractions with the product. For example, a user can interact withclient system 110 to request repositioning or reorientation of theproduct in the image. Client system 110 can be configured to provide therequest to application system 130, which can generate instructions toreposition or reorient the product in the image. For example, theinstructions can include an updated version of the image with theproduct repositioned or reoriented in accordance with the requestprovided by client system 110.

Network 140 can facilitate communication and sharing of informationbetween client system 110, host system 120, and application system 130.Network 140 may be any type of network that provides communications,exchanges information, and/or facilitates the exchange of information.For example, network 140 may be the Internet, a Local Area Network, acellular network, a public switched telephone network (“PSTN”), or othersuitable connection(s) that enables transmission of information betweenthe components of system 100. Network 140 may support a variety ofelectronic messaging formats and may further support a variety ofservices and applications for communicating between client system 110and host system 120, or client system 110 and application system 130.

FIG. 2 depicts a method 200 for initializing a visualizationapplication, in accordance with disclosed embodiments. A webpagereceived from host system 120 and displayed by client system 110 can bemodified using data and instructions received from application system130 to display the visualization application. In this manner, thevisualization application can be integrated into webpage provided by ahost website, allowing the user to visualize products without having toleave or reload the website. In this manner, the visualizationapplication can modify a webpage (which need not originally supportsingle-page application functionality) to provide single-pageapplication functionality. Furthermore, method 200 can use a script toautomate integration of the visualization application into the webpage.By using such a script, maintenance and development of the visualizationfunctionality can be separated from maintenance and development of thewebpage. In some embodiments, integration of the visualizationfunctionality into the webpage may then require minimal modification ofthe webpage (e.g., adding a script tag, as described herein).

In step 210, client system 110 can obtain a webpage from host system120. Client system 110 can obtain the webpage through a networkconnection established with host system 120 according to networkprotocol (e.g., using an HTTP request, or the like). Host system 120 canprovide the webpage in response to a request received from client system110. The disclosed embodiments are not limited to a particular method ofobtaining the webpage from host system 120.

The webpage can include code and instructions (e.g., Hypertext MarkupLanguage or the like) for displaying content in a browser of clientsystem 110. In some embodiments, the webpage can include productinformation for at least one product. For example, the webpage can be aproduct page providing detailed product information about a singleproduct, or a catalog page including product information about each ofmultiple products. As a non-limiting example, the product informationcan include product name, product brand, product category, modelnumbers, serial numbers, stock keeping units (SKUs), product coloroptions, pricing information, product dimensions, product availabilityand inventory information, product installation information, or similarinformation about the product

The webpage can include vendor product code information usable toassociate a product with visualization information for the product. Insome embodiments, vendor product code information can be scrapable fromthe webpage. In various embodiments, the visualization information canbe associated with an identifier for the product. The identifier may beunique for a combination of product and host system 120 (or productprovider associated with host system 120), or unique for a combinationof product, product type, and host system 120 (or product providerassociated with host system 120).

Vendor product code information included in the webpage can be used todetermine, create, or recover the identifier, enabling retrieval of thevisualization information. For example, a stock keeping unit (SKU)number or Universal Product Code (UPC) for a product can serve as anidentifier, associating the product with visualization information. Thewebpage can include the SKU number for the product. As an additionalexample, an identifier can be created (e.g., using one or more stringoperations) from two or more items of product information for a product(e.g., by concatenating the product name, product brand, and productcolor; or the like). The webpage can include the two or more items ofproduct information for a product, enabling the identifier to berecreated by scraping the two or more items of product information fromthe webpage.

In step 220, client system 110 can obtain a script from applicationsystem 130. Client system 110 can obtain the script through a networkconnection established with application system 130 according to networkprotocol (e.g., using an HTTP request, or the like). The disclosedembodiments are not limited to a particular method of obtaining thewebpage from application system 130.

In some embodiments, application system 130 can provide the script inresponse to a request received from client system 110. Client system 110can generate the request while processing the webpage. For example, thewebpage can include instructions (e.g., an HTML <script> tag) that, whenparsed or executed by client system 110, cause client system 110 torequest the script from application system 130. In some embodiments, inaddition to the location of the script, the instructions can indicate atype, language, or version of the script (e.g., JavaScript or anothersuitable scripting language), or other information used in parsing orexecuting the script by client system 110. The disclosed embodiments arenot limited to a particular scripting language or script format.

In some embodiments, the script can be specific to at least one of thewebpage, host system 120, a product provider associated with host system120, or a product type or category. In various embodiments, applicationsystem 130 can provide a previously generated script. In someembodiments, application system 130 can provide a script generated inresponse to the request. A script generated in response to the requestcan be specifically tailored to each request by client system 110.

Although depicted in FIG. 2 as two separate steps, in some embodiments,steps 210 and 220 can be performed as one step. For example, host system120 can store a cached copy of the script, mirroring the script storedin the application system 130, and transmit the copy of the script toclient system 110 with the webpage or after the website is transmitted.

In step 230, according to the script obtained in step 220, client system110 can obtain identifiers for each of any products described orreferenced in the webpage. In some embodiments, the script can configureclient system 110 to scrape or parse the webpage for vendor product codeinformation. For example, client system 110 can be configured to extractvendor product code information, as described herein, from a DocumentObject Model (DOM) of the webpage. In some embodiments, the vendorproduct code information can be stored in a predetermined location inthe webpage (e.g., at a known location within a particular documentelement, within a hidden or displayed form element, within an elementwith a known name or id, or the like). In some embodiments, the vendorproduct code information can be extracted from the URL of the webpage.The script, which can be specific to the webpage, can cause clientsystem 110 to extract the vendor product code information from thepredetermined location in the webpage. In various embodiments, thescript can extract the product information by searching the source codeof the webpage for text with predetermined characteristics (e.g.,product names, descriptions, colors, serial numbers, model numbers, orthe like). In some embodiments, such searching can be performed usingregular expressions or the like. The disclosed embodiments are notlimited to embodiments in which client system 110 obtains allidentifiers for all products described or referenced in the webpage. Aswould be appreciated by those of skill in the art, client system 110 mayonly attempt or be able to obtain identifiers for each of a subset ofthe products described or referenced in the webpage.

In some embodiments, client system 110 can be configured by the scriptto generate the identifier from the vendor product code information. Forexample, the identifier can be generated by applying one or more stringoperations to items of the vendor product code information (e.g.,concatenating product name, brand, model number, and color to generatean identifier for the product).

In some embodiments, the product information can be included in the URLrequesting the script as a parameter or part of the address. In theseinstances, the product information may have been transmitted toapplication system 130 prior to receipt of the script by client system110. Application system 130 can then transmit a script that has beentailored to match the product.

In some embodiments, client system 110 can be configured by the scriptto manage user identity or session information. For example, in someembodiments, client system 110 can be configured by the script to placeor check cookies identifying the user or establishing a session. Tocontinue this example, when client system 110 lacks a current usersession cookie, client system 110 can be configured by the script togenerate such a cookie (e.g., by requesting a session or user identifierfrom application system 130 and storing the obtained information in thecookie). This cookie may be, or may be treated as, a first party cookieby client system 110. For example, the cookie can be associated with adomain of host system 120. Accordingly, this cookie may not be blockedby ad blockers or third-party cookie blockers. If a current user sessioncookie does exist in client system 110, but application system 130 doesnot recognize a corresponding session, client system 110 can beconfigured by the script to request application system 130 create a newuser session associated with the user or with client system 110.

In step 240, according to the script obtained in step 220, client system110 can query application system 130 for visualization informationcorresponding to the identifier(s) obtained in step 230. The query, andthe response to the query, may not cause client system 110 to reload thewebpage. For example, the query can be transmitted to application system130 using Asynchronous JavaScript+XML (AJAX) techniques, or the like,and the data transmitted with the query can be organized in the form ofJavaScript Object Notation (JSON), or the like. As an additionalexample, the query can be as simple as a URL pointing to a predeterminedlocation in application system 130. The visualization information may bepresent at that location.

In response to the query, application system 130 can determine whethervisualization information corresponding to the identifier(s) isavailable. In some embodiments, application system 130 can compare theidentifier(s) to a list or database of products having visualizationinformation. Application system 130 can transmit to client system 110 anindication of whether visualization information corresponding to theidentifier(s) is available.

In step 250, when visualization information is available for a productand according to the script obtained in step 220, client system 110 canmodify the webpage to display a control. The control can be aninteractive element in the webpage, for example, a button, a toggle, orthe like. In some embodiments, the control may have been included in thewebpage received from host system 120, but may be inactive (e.g.,disabled, hidden, or the like). When the query indicates thatvisualization information is available for the product, client system110 can modify the webpage to activate the control (e.g., permitting thecontrol to be interacted with by a user). In some embodiments, thecontrol may not have been included in the webpage received from hostsystem 120. The control may then be added to the webpage. For example,client system 110 can be configured by the script to modify the webpageto include a selectable button at a location associated with theproduct. As the script may be specific to the webpage or to host system120, the modification can preserve the look and feel of the webpage.Thus the control can be displayed in a manner that appears natural tothe user and without disrupting the user experience.

In step 260, when visualization information is available for a productand according to the script obtained in step 220, client system 110 canretrieve a visualization application from application system 130. Insome embodiments, client system 110 can retrieve a visualizationapplication in response to selection of the control (e.g., by a userinteraction) in the modified webpage. For example, selection of thecontrol can trigger an event. The script can handle the event, causingclient system 110 to retrieve the visualization application fromapplication system 130. For example, when the webpage is modified todisplay a button control, a selection of the button by a user can causeclient system 110 to retrieve the visualization application.

In step 270, when visualization information is available for a productand according to the script obtained in step 220, client system 110 canmodify the webpage to display the visualization application. Thevisualization application can enable the user to select an image andmodify the image to display the product, as described herein. In someembodiments, client system 110 can be configured by the script to modifythe webpage to include a container, such as an iframe, or the like.Client system 110 can then load the retrieved visualization applicationinto the container. The disclosed embodiments are not limited to use ofa container. Other modifications resulting in display of thevisualization application are also envisioned, including addition of alayer over the existing webpage, opening a new tab or a new window todisplay the visualization application, or removing or rearranging one ormore webpage elements to create space for displaying the visualizationapplication.

The operations of method 200 have been labeled and describedsequentially, for convenient description. As would be appreciated bythose of skill in the art, the indicated sequence is not intended to belimiting. Steps may be rearranged or combined, or additional steps maybe added. For example, in some embodiments, a single operation caninclude querying product availability (step 240) and obtaining thescript (step 220). For example, the webpage obtained in step 210 cancontain an URL pointing to the location of the script and at least oneparameter can be combined with the URL, in effect querying availabilityof visualization information for the product. Similarly, thevisualization application can be retrieved prior to selection of thecontrol in the modified webpage. The visualization application may notbe integrated into the webpage or may be integrated into the webpage butset to “invisible” or “inactive,” until the control is selected.

FIG. 3 depicts a method 300 for interacting with a visualizationapplication, in accordance with disclosed embodiments. Such interactionscan occur after modification of the webpage to display the visualizationapplication. In some embodiments, a user can interact with thevisualization application (e.g., through a graphical user interface ofclient system 110) to provide an image. The image can then be modifiedto include a product. The user may then further interact with thevisualization application to modify the display of the product in theimage (e.g., by modifying the object's placement, orientation, or thelike).

In step 310, a user can interact with client system 110 to select animage. The image can be of a location (indoor or outdoor) where theproduct can be placed. As a non-limiting example, the image can be of aroom. The user can interact with a graphical user interface provided bythe visualization application to select the image. In some embodiments,the graphical user interface may provide options for uploading an image.For example, the visualization application may provide the option toupload images from a memory accessible to client system 110. As afurther example, the visualization application may provide the option toacquire an image and upload the acquired image (e.g., visualizationapplication may provide an option to access a camera of client system110 or communicatively connected to client system 110, to capture animage for uploading). To continue this example, when client system 110is a smartphone, the visualization application can be used to obtain animage using the camera of the smartphone. In various embodiments, thegraphical user interface of the visualization application may provideoptions for selecting an existing image (e.g., an image previously savedor uploaded by the user, or a default image). In some embodiments,client system 110 can be configured to communicate with applicationsystem 130 to obtain the image options for selection. For example,application system 130 can retrieve the image options and provide themto client system 110 for display. As an additional example, applicationsystem 130 can provide information enabling client system 110 toretrieve image options from another location. The image can be a JPG,GIF, Bitmap, PNG, or other suitable format. The disclosed embodimentsare not limited to any particular standard or format of the image. Anexemplary graphical user interface is described herein with regards toFIGS. 4A to 4L.

As described above, in some embodiments the user can cause client system110 to display the visualization application by selecting a controlassociated with a product. The visualization application can beconfigured to display this product in the image. However, in someembodiments, the user can interact with the visualization application toselect a product for placement in the image in addition to selecting theimage. For example, the visualization application can display productsavailable for visualization to the user and the user can interact withthe visualization application to select a product. In such embodiments,the visualization application can display a thumbnail of each productand other information that may help the user to determine which productto select, (e.g. price, material, rate, etc.).

In step 320, client system 110 can provide a request to display theselected product in the selected image to application system 130. Therequest can be provided using network 140. The disclosed embodiments arenot limited to any particular method or protocol for providing therequest. In some embodiments, when the image is uploaded by clientsystem 110, the request can include the image. In various embodiments,when an image option is selected, the request may include the image oran indication of the image. In some embodiments, client system 110 canconvert the selected image into a predetermined format beforetransmitting it to application system 130, while in some otherembodiments, client system 110 can transmit the selected image toapplication system 130 without converting the image. The disclosedembodiments are not limited to a particular format for transferring theimage. In some embodiments, the request can indicate the selectedproduct. In various embodiments, the request can include metadataconcerning the image. For example, depth information can be associatedwith the image. In some embodiments, client system 110 can capture thedepth information using one or more images from multiple cameras ormultiple images from the same camera. The disclosed embodiments are notlimited to a particular method of acquiring such depth information.

In step 330, application system 130 can generate instructions formodifying the selected image to display the selected product. In someembodiments, application system 130 can generate instructions enablingclient system 110 to update the display of the product in the selectedimage in response to user input. In such embodiments, the instructionscan include a product model and information enabling display of theproduct in the selected image. For example, when the product is a lamp,the instructions can include a three-dimensional model (3D) of the lampand information specifying the perspective of the image and the distancefrom a viewpoint of the image to one or more surfaces depicted in theimage. Such information could enable client system 110 to scale androtate the 3D model of the lamp for correct placement on the one or moresurfaces depicted in the image. This may help to ensure that theperspective and/or scaling of the product displayed in the selectedimage is consistent with the perspective and scaling of the selectedimage.

In various embodiments, the instructions received from applicationsystem 130 may not enable client system 110 to update the display of theproduct in the selected image in response to user input. Instead, clientsystem 110 may request updates to the modified image from applicationsystem 130. In some embodiments, such instructions can include a versionof the selected image modified to display the product in a particularposition and orientation.

In some embodiments, generating the instructions can include creating a3D model from the image. The 3D model provides a model of the scenecaptured in the image, including objects present in the image. Thisinformation enables the perspective and scaling of the image and allobjects therein to be varied. Application system 130 can be configuredto generate the 3D model using a machine learning model, such as aconvolutional neural network. The machine learning model can beconfigured to estimate depth and perspective information for the image.For example, the machine learning model can be configured to estimate aviewpoint for the image and a distance from the viewpoint to a depictedsurface in the image or pixel in the image. In various embodiments, whenthe image depicts a room, the machine learning model can be configuredto determine a shape of the room (e.g., the location of the walls,floor, or ceiling in the image, with respect to the viewpoint or to eachother, or the like). In some embodiments, when the request received fromclient system 110 includes depth information, Application system 130 canbe configured to use such depth information when creating the 3D model.

In some embodiments, application system 130 can be configured to performsemantic segmentation of the image. Such semantic segmentation canidentify portions of the image relevant to placement of a product in theimage. Semantic segmentation may be used to identify image pixelsassociated with a shared object represented in the image, therebyenabling objects represented in the image to be identified. As anon-limiting example, semantic segmentation can be used to identifyfloors, walls, countertops, items of furniture (e.g., tables, bookcases,shelving), or the like in the image. On the basis of the identifiedobjects represented in the image, portions of the image may bedesignated for placement of the product in the image. In someembodiments, the semantic segmentation can be performed using, inaddition to the image, the type of the product or the 3D model generatedfrom the image. For example, when the product is a type of flooring,this product type can inform the semantic segmentation (e.g., throughselection of a machine learning model for performing the segmentation oras an input to the machine learning model that performs thesegmentation). Similarly, the 3D model can inform the semanticsegmentation (e.g., the 3D model can indicate surfaces that maypotentially be part of a floor). This may help to reduce processingoverhead, enabling a targeted semantic segmentation of the image to becarried out, in which the image is analyzed for the presence of specificobjects. The type of product selected for visualization may be used toidentify one or more objects present in the image required by theproduct to provide its function. For example, where the selected productfor visualization relates to a type of flooring, the semanticsegmentation may be restricted to identifying image objectscorresponding to a floor in the image. The floor being essential to theflooring's function.

Application system 130 can use one or more machine learning models toperform the semantic segmentation of the image. For example, Applicationsystem 130 can perform semantic segmentation using a convolutionalneural network built on an AlexNet, VGG-16, GoogLeNet, ResNet, or othersuitable architecture. The convolutional neural network can be trainedto perform region-based semantic segmentation, fully convolutionalnetwork-based semantic segmentation, or to perform another suitablemethod of semantic segmentation.

In some embodiments, application system 130 can be configured to usemultiple machine learning models to perform semantic segmentation of theimage. The multiple machine learning models can be trained to identifydiffering portions of the image. For example, one machine learning modelcan be trained to identify floors in the image, while another machinelearning model can be trained to identify walls in the image.Application system 130 can be configured to determine which of themultiple machine learning models to use based on a type of the product.For example, when the product is a wall clock, application system 130can perform semantic segmentation on the image using a machine learningmodel trained to identify walls in the image. As an additional example,when the product is flooring, application system 130 can performsemantic segmentation on the image using a machine learning modeltrained to identify floors in the image.

The disclosed embodiments are not limited to embodiments using machinelearning models trained to identify a single semantic class. Forexample, the disclosed embodiments can be configured to use a machinelearning model trained to identify walls, floors, and counters in theimage, or more generally identify surfaces in the image.

In some embodiments, application system 130 can be configured to storethe image and the 3D model of the image. Should the user later want toplace another product in the stored image, the stored image and modelcan be retrieved and reused.

Application system 130 can be configured to maintain a data storagesubsystem (e.g. a database) storing product models. In response to arequest from client system 110 indicating a selected product,application system 130 can be configured to retrieve a model for theselected product. The model of the selected product can include spatialinformation and surface detail information. The spatial information canspecify the dimensions of the product. In some embodiments, such spatialinformation can include a mathematical representation of one or moresurfaces of a product. In various embodiments, the spatial informationcan comprise a 2D or 3D model of the product. For example, a 3D model ofa lamp can specify the location of points on the surface of the lamp ina three-dimensional space. As a further example, a 2D model of a postercan specify the height and width of the poster. Surface detailinformation can include textures, colors, patterns, or the like. Forexample, a product can be associated with a texture mapping, which canbe applied to a 2D or 3D model of the product, or to a surface in the 3Dmodel of the image. In various embodiments, some products may not beassociated with a spatial model. For example, in some embodiments,products lacking predetermined dimensions, such as flooring, molding,paneling, wallpaper, countertops, or the like, may be associated withsurface detail information but not spatial information.

Application system 130 can, in some embodiments, be configured toretrieve a product model for the selected product and include theproduct model in the instructions. Application system 130 canalternatively or additionally modify the selected image using theproduct model and then include the modified image in the instructions.

Application system 130 can modify the selected image by rendering theproduct model at a particular location in the selected image. Applicantsystem 130 can be configured to use a suitable rendering method, such aspolygon-based rendering, scanline rendering, ray tracing, rasterization,or the like. Such rendering can include determining pixels in theselected image obscured by the rendered product model and replacingthese pixels with corresponding pixels of the rendered product model.For example, application system 130 can determine that a wall clock maycover an area of the wall and replace the pixels in the selected imagewith corresponding pixels in a rendering of the wall clock. In variousembodiments, the particular location can be determined based oninformation included in the request. For example, the user can specify alocation for a product (e.g., a lamp) in the image (e.g., on a tabledepicted in the image). In some embodiments, the particular location canbe determined by application system 130. For example, application system130 can determine suitable locations in the image based on semanticsegmentation, as described herein, and a type of the product. In thisexample, application system 130 may determine that the product is a lampand that the image includes a surface suitable for placement of a lamp(e.g., a tabletop, countertop, shelf, or the like). Application system130 may then render the lamp on the suitable surface.

Application system 130 can modify the selected image by painting animage of the product on suitable portions of the selected image, usingsurface detail information for the product and a 3D model of the image.As described above, application system 130 can perform semanticsegmentation on the image to associate classes with pixels in the image.Within the present context, class may be used to designate the type ofobject present in the image. In some instances, when a type of theproduct matches one of the classes, an image of the product generatedusing the surface detail information can be painted over pixelsassociated with that class. For example, when the product is a type offlooring, application system 130 can replace portions of the selectedimage associated with the class “floor” with correspondingtexture-mapped portions of the flooring.

In step 340, application system 130 provides the instructions generatedin step 330 to client system 110. The instructions can be provided usingnetwork 140. The disclosed embodiments are not limited to any particularmethod or protocol for providing the instructions. In some embodiments,the instructions may include a modified image displaying the selectedproduct. In various embodiments, the instructions may include theselected image (e.g., when the request in step 320 indicated an imageoption but did not include an uploaded image) and a rendered version ofa product model (e.g., a rendered version of the product model,appropriately scaled and oriented such that it can be combined with theimage, together with information enabling the client system to combinethe images). In some embodiments, the instructions can include a productmodel for the selected product, together with instructions for renderingthe product model and displaying the rendered product model in theselected image. For example, the instructions can include a 3D model ofthe selected image and results of semantic segmentation of the selectedimage (e.g. associations between pixels in the selected image).

In step 350, client system 110 can be configured to display the selectedimage, modified to include the selected product, using the receivedinstructions. When the instructions include a modified image displayingthe selected product, client system 110 can be configured to display themodified image. When the instructions include a rendered product modeland the selected image, client system 110 can be configured to combinethe rendered product model and the selected image to generate themodified image. Client system 110 can then display the modified image.When the instructions include the product model and instructions forrendering the product model and displaying the rendered product model inthe selected image, client system 110 can perform such rendering anddisplay the resulting modified image. For example, the instructionsinclude a 3D spatial model of the product, surface detail informationfor the product, and a 3D model of the image. The 3D model of the imagecan include perspective information and depth information for surfacesin the image. Client system 110 can be configured to use the perspectiveand 3D model of the image to render the product in the image atlocations indicated by the user.

Optionally, in step 360, the client system 110 can be configured toupdate the modified image based on user interactions with thevisualization application. Preferably, the user can interact with themodified image to adjust the location and orientation of the displayedproduct in the image, without such interaction requiring loading of thewebpage. For example, the user can rotate the product and move theproduct to place the product at a different location. Preferably, clientsystem 110 can limit the range within which the product can be placeddepending on the type of the product and the classification of pixels inthe selected image. The method may comprise identifying pixels in theselected image having a classification associated with the type ofproduct, and restricting placement of the product within the image tolocations corresponding with the identified pixels having theclassification associated with the type of product. For example, theuser can move a carpet within the range of pixels identified asrepresenting floors. In some embodiments, client system 110 canre-render the product image or/and re-combine the product image with theselected image as described above. In some embodiments, client system110 can transmit the user interaction to application system 130 andrequest display of the product in the room image with updatedinstructions. In these cases, the system can repeat steps 330 to 350.

FIG. 4A-L depicts an exemplary client system 110 implementing anembodiment of the current invention. FIG. 4A illustrates the interfaceof client system 110 when visiting host system 120. For the purpose ofthe illustration, host system 120 is a website for a home suppliesstore. Client system 110, as described above, can be any computingdevice with a display. For the purpose of illustration, client system110 in this example is a smartphone with a camera. Client system 110requests a webpage from the host system 120 (e.g. a website of the homesupplies store) and displays the webpage to the user. The webpage canlist a group of categories of products (e.g. floors, wallpapers,furniture, etc.). Each of the listed categories is associated with abutton labeled with the name of the category, which, if clicked, causesclient system 110 to request the webpage depicted in FIG. 4B.

FIG. 4B depicts an exemplary webpage retrieved from host system 120.This webpage shows a list of products. Similar to FIG. 4B, FIG. 4Cdepicts an exemplary webpage retrieved from host system 120. Unlike FIG.4B, FIG. 4C depicts product information for a single product. Duringloading of the webpages depicted in FIGS. 4B and 4C, client system 110requested a script from application system 130. As described above withregards to FIG. 2, the script has executed, causing client system 110 toidentify the products listed in each of the webpages. Client system 110has communicated with application system 130 to determine thatvisualization information for the products is available. Accordingly,each webpage has been modified to include “See in My Room” buttonsbeneath each product. For each webpage, the user can click the button tostart the visualization application.

FIG. 4D and 4E depict interactions with a visualization applicationfollowing selection of a “See in My Room” button beneath a product. InFIG. 4D, the webpage has been modified to include the visualizationapplication. The visualization application is providing an interface forselecting either a predetermined image (e.g. “cozy space”) or choosingto upload an image (e.g., “Take Photo of My Room”). When the userselects uploading an image, client system 110 can operate the camera ofclient system 110 to enable the user to take a photo of the room. Insome embodiments, although not depicted in the FIG. 4D and 4E, the usercan select an image previously stored by application system 130 or animage previously obtained by client system 110. After the photo isselected or confirmed, client system 110 can transmit the photo or anindication of a selected photo to application system 130, which, inreturn, generates instructions and transmits them to client system 110as described above with regards to FIG. 3.

FIG. 4F depicts display, by the visualization application, of a modifiedimage displaying a product. In this non-limiting example, the floor inthe selected image (depicted in FIG. 4E) has been replaced with an imageof a flooring product (the hardwood floor depicted in FIG. 4C). Asdescribed herein, the “floor” pixels in the selected image have beenidentified through semantic segmentation. The surface detail information(e.g., the pattern, color, and texture of the flooring product) of theproduct have been texture mapped to the identified pixels.

FIG. 4G-4I depicts an interface for updating the modified image. In FIG.4G, a list of thumbnails of suggested products is displayed in the userinterface. In some embodiments, client system 110 can be configured tostore a list of such products. Alternatively, client system 110 canquery application system 130 to populate the list of products. AlthoughFIG. 4G depicts the thumbnails at the bottom of the display, thethumbnails can be placed at any portion of the display. As depicted inFIG. 4H, the user can search for a product by entering text into clientsystem 110. As depicted in FIG. 4I, the user can apply one or morefilters to limit the number of products being displayed. In someembodiments, client system 110 can search or filter the product locally.In some embodiments, client system 110 can transmit the search requestor the filtering request to application system 130 and receive fromapplication system 130 an updated product list. Following selection of anew product, the modified image can be updated to display the newproduct. Such updating can include providing, by client system 110 toapplication system 130, a request to display the new product in theselected image. Client system 110 can receive instructions fordisplaying the new product in the selected image in response.

FIGS. 4J-4L depicts interactive modification to the display of a productin an image. In this non-limiting example, the product is a rug. In FIG.4J, the rug is displayed in a first position on the floor of the room.The view of the rug is adjusted to match the perspective and viewpointof the image. Furthermore, the existing ottoman is displayed as being ontop of the rug. In FIG. 4K, the visualization application providesinstructions on changing the position and orientation of the rug in theimage using a set of pre-defined touch screen gestures (e.g., press anddrag to move the object around the room and press and turn with twofingers to rotate the object). In some embodiments, the user caninteract with the object using a mouse, a keyboard, or any otheravailable input device. As depicted in FIG. 4L, in response with theuser interaction, the displayed position and orientation of the objectcan be updated. As described above with regards to FIG. 3, such updatingcan be performed locally by client system 110, using a 3D model of theimage and a product model, performed remotely at application system 130,or performed in part locally and in part remotely. For example, clientsystem 110 can transmit the user interactions to application system 130,which can modify the images and transmit the modified image back toclient system 110. Alternatively, client system 110 can transmit theproduct selection to application system 130 and receive a product modeland instructions for positioning the product in the selected image.Client system 110 can then render the product model, combine therendered product model with the selected image, and display theresulting modified image to the user without communicating with theApplication System 130.

FIG. 5 depicts a schematic of exemplary computing system 500 forperforming the envisioned systems and methods, consistent with disclosedembodiments. In some embodiments, computing system 500 can include aprocessor 510, memory 515, display 520, I/O interface(s) 525, andnetwork adapter 530. These units may communicate with each other via bus535, or wirelessly. The components shown in FIG. 5 may reside in asingle device or multiple devices.

Consistent with disclosed embodiments, processor 510 may comprise acentral processing unit (CPU), graphical processing unit (GPU), orsimilar microprocessor having one or more processing cores. Computingsystem 500 may include one or more processors 510 and may furtheroperate with one or more other processors that are remote with respectto processors 510. Memory 515 may include non-transitory memorycontaining non-transitory instructions, such as a computer hard disk,random access memory (RAM), removable storage, or remote computerstorage. In some aspects, memory 515 may be configured to store data andinstructions, such as software programs. For example, memory 515 may beconfigured to store data and instructions. In some aspects, processor510 may be configured to execute non-transitory instructions and/orprograms stored on memory 515 to configure computing system 500 toperform operations of the disclosed systems and methods. In variousaspects, as would be recognized by one of skill in the art, processor510 may be configured to execute non-transitory instructions and/orprograms stored on a remote memory to perform operations of thedisclosed systems and methods.

Display 520 may be any device which provides a visual output, forexample, a computer monitor, an LCD screen, etc. I/O interfaces 525 mayinclude hardware and/or a combination of hardware and software forcommunicating information to computing system 500 from a user ofcomputing system 500, such as a keyboard, mouse, trackball, audio inputdevice, touch screen, infrared input interface, or similar device.Network adapter 530 may include hardware and/or a combination ofhardware and software for enabling computing system 500 to exchangeinformation using external networks, such as network 140. For example,network adapter 530 may include a wireless wide area network (WWAN)adapter, a Bluetooth module, a near field communication module, or alocal area network (LAN) adapter.

FIG. 6 depicts exemplary method 600 for visualization of products in adisplay interface, in accordance with disclosed embodiments. As depictedin FIG. 6 and discussed herein, the disclosed embodiments are notlimited to displaying products in an image using a visualizationapplication. In some embodiments, application system 130 can providegraphic resources, as described herein, to client system 110 for use bynative tool 601 in displaying the product in the image data. A user canthen interact with the client system 110 to modify the display of theproduct in the image (e.g., by modifying the product's placement,orientation, or the like),In such embodiments, client system 110 neednot obtain the visualization application. Furthermore, client system 110need not provide an image to application system 130, a simplificationthat may improve the user experience. Client system 110 may be able todisplay the product in the image data as the image data is acquired,further improving the user experience. In some instances, for example, auser of client system 110 may be empowered to move around an environmentand view a virtual image of the product from different angles orperspectives in the environment. Additionally, a more accuraterepresentation can be provided, as the image data may provide moreinformation about the environment than a single image. Furthermore,native tool 601 may have access to additional sources of data (e.g.,accelerometers in client device 110, or the like) for determining thelayout of the environment, or may be more powerful than a practicablydownloadable visualization application.

Consistent with disclosed embodiments, the image data can includepictures, video data, a live stream, or the like. The image data can beobtained by client system 110 from an imaging device (e.g., a digitalcamera capable of capturing photographs, recording video, providing thelive stream of images, or the like). In some embodiments, the image datacan be retrieved from a memory accessible to client system 110 orreceived from another system or device (e.g., a security camera,camcorder, webcam, or the like). In some embodiments, method 600 canenable a real-time display of the product in the image data.

Native tool 601 may include a graphic rendering programming or engine.Native tool 601 can be or include one or more augmented reality tools(e.g., ARKit, ARCore, or the like). Native tool 601 can be incorporatedinto a mobile web browser (e.g., Safari, Chrome, or the like) or amobile application (e.g., an app) of client system 110. Native tool 601can use the graphic resources received by client system 110 fromapplication system 130 to generate augmented reality (AR) or virtualreality (VR) media displaying the product in image data acquired byclient system 110. In some embodiments, native tool 601 can beconfigured to display the augmented reality (AR) or virtual reality (VR)media in an application embedded in a webpage (e.g. the webpagecontaining the control selected by the user to start the visualization).In some embodiments, native tool 601 can be configured to display theaugmented reality (AR) or virtual reality (VR) media in a separateapplication (e.g., separate from the webpage containing the controlselected by the user to start the visualization).

In some such embodiments, client system 110 can be or include acomputing device such as a terminal or interactive kiosk machine. Theterminal or interactive kiosk machine can be configured to supportnative tool 601.

In some embodiments, method 600 can include steps of obtaining a webpage(e.g., from host system 120), obtaining a script (e.g., from applicationsystem 130), obtaining identifiers for each of at least some of theproducts described or referenced in the webpage, and querying forproduct availability. Such steps can be performed as discussed hereinwith respect to steps 210 to 240 of FIG. 2. In some embodiments, whenvisualization information is available for a product and according tothe obtained script, client system 110 can modify the webpage to displaya control for the product. As described above with regards to step 250of method 200, the control can be an interactive element. Displaying thecontrol may include activating the control.

In step 610 of method 600, a user can interact with the modified webpageto select a product (e.g., by selecting the control corresponding to theproduct). The disclosed embodiments are not limited to any type ofinteraction. In some embodiments, the user can interact with themodified webpage as described above with regards to step 260 of method200.

In step 620, in response to the interaction in step 610, client system110 can obtain graphic resources from application system 130. In someembodiments, client system 110 can provide a request to applicationsystem 130. In some embodiments, the request can be provided usingnetwork 140. The disclosed embodiments are not limited to any particularmethod or protocol for providing the request. The request can indicatethe product for which the graphic resources are required (e.g., it caninclude a value or other indicator of the product; be directed to a URLor the like specific to the product; or the like). In some embodiments,the request can provide information indicating a type, model, version,or the like of native tool 601. In various embodiments, the request canspecify a particular type of graphic resource.

In response to the request, application system 130 can provide thegraphic resources to client system 110. In some embodiments, applicationsystem 130 can retrieve the graphic resources from a memory accessibleto application system 130. In various embodiments, application system130 can receive the graphic resources from another system (e.g., asystem associated with a manufacturer, supplier, or distributor of theproduct, or the like). In some embodiments, application system 130 cancreate the graphic resources in response to the request. In someembodiments, application system 130 can provide graphic resourcesselected based on at least one of the selected product or theinformation indicating the native tool of client system 110. In someembodiments, application system 130 can generate the graphic resourcesbased on the received request. In some instances, for example, based onthe information regarding the native tool of client system 110,application system 130 can customize graphic resources to be compatiblewith the native tools of client system 110 (e.g., a USDZ file formatwhen native tool 601 is ARKit or the like; or an OBJ/MTL file formatwhen native tool 601 is ARToolkit or the like).

In step 630, a browser of client system 110 can call or invoke nativetool 601. In some embodiments, the script received from applicationsystem 130 can cause the browser to provide commands to native tool 601(e.g., using an API exposed by native tool 601). The commands can, insome embodiments, include commands to acquire image data. The commandscan, in various embodiments, include commands to display the product ata location in an environment depicted in the image data. The commandscan include commands to start an application for displaying the productin the environment. In some embodiments the application can be embeddedin the browser. In various embodiments, the application can be separatefrom the browser.

In step 640, client system 110 can display the product in obtained imagedata. The display can be or include an AR/VR representation of theselected product. In some embodiments, the native tool of the clientsystem 110 can generate the display by imposing, combining, or otherwisecausing the received graphic resources to appear on or at a location inthe environment. For example, a 3D representation of a spatial model ofthe product can appear in a camera feed in real-time, viewable on thedisplay of the client system 110. The client system 110 can beconfigured to update the AR/VR representation of the product based onuser interactions with client system 110. For example, the user caninteract with the AR/VR representation of a displayed product to adjustthe location and orientation of the displayed product in theenvironment. For example, the user can rotate the product and move theproduct to place the product at a different location in the environment.In some embodiments, the displayed portion of the environment can changewhen the camera of client system 110 shifts, moves, rotates, orotherwise alters its field of view. In such instances, the native toolof client system 110 can fix the displayed product at a location in theenvironment regardless of the changes in the field of view.

In some embodiments, client system 110 can display an AR/VR interface inthe environment to enable the user to update the display in the capturedenvironment. In some embodiments, the AR/VR interface can enable theuser to select or update a selection of a product for display in thecaptured environment. In various embodiments, the AR/VR interface canenable the user to modify characteristics of the product displayed inthe captured environment (e.g., modifying at least one of a color,shape, size, composition or material, or the like). The AR/VR interfacecan be generated to be displayed in conjunction with the environmentcaptured in step 650. For example, the native tool of client system 110can generate a control interface in the live camera view displayed,enabling the user to interact with the control interface while viewingthe camera feed in real-time. In some embodiments, the AR/VR interfacecan indicate a list of products available for display in theenvironment. In some instances, the available products can berepresented by graphics or symbols, such as icons, alphanumeric symbols,or a small representative images (such as a thumbnail image). The listof available products can be presented in a menu-like interface (such asa drop-down menu, a scroll menu, or other similar interfaces).

The disclosed embodiments are not limited to using an identifier of aproduct generated from information extracted from the webpage. In someembodiments, the product to be displayed can be randomly selected byapplication system 130 based on product availability information. Forexample, when a user selects (e.g., clicks on) a visualization controlof the visualization application, the visualization application canstart with a default or a random product before receiving user selectionof product (e.g., step 320). As a non-limiting example, a banner on awebsite of host system 120 can display the label “Visualizer”. When auser selects the banner, an instance of the visualizer can start with adefault or random product. The default product can be a predeterminedproduct chosen from among the available products. The random product canbe a product chosen from among the available products by a random orpseudo-random algorithm. In some embodiments, the banner can be aselectable advertisement for the product.

The disclosed embodiments are not limited to requesting scriptsassociated with host system 120. In some embodiments, the script may notbe specifically associated with host system 120, but can instead be ageneric script. Application system 130 can adapt the generic script tohost system 120. For example, when a user interacts with thevisualization application, the generic script can be customized to hostsystem 120 based on a context associated with host system 120, such asthe domain of the webpage (e.g., obtained from the current URL), or thecontents of the webpage. As a non-limiting example, a generic script canbe provided to client system 110. In some embodiments, the genericscript can be configured by application system 130 based on a context ofthe webpage accessed by client device 110. When executed, the configuredscript can modify the web page of client system 110 to include aninteractive graphical indicator. The graphical indicator can beselectable to start an instance of the visualization application. Theinstance of the visualization application can provide contentappropriate for the webpage, based on the configuration of the scriptperformed by application system 130. In various embodiments, the genericscript can be sent to the client system 110. When the visualizationapplication is started, it can provide the contextual information to theapplication system 130, which can the provide appropriate customizationinformation. For example, if host system 120 is associated with awebsite for a furniture retailer, application system 130 can determinethat the context of host system 120 is a furniture retailer (or is thatparticular furniture retailer), and a generic script configured forfurniture retailers (or for that particular furniture retailer) can beprovided to client system 110. If host system 120 is associated with awebsite for a carpet retailer, application system 130 can determine thatthe context of host system 120 is a carpet retailer (or is thatparticular carpet retailer), and the generic script, configured forcarpet retailers (or that particular carpet retailer) can be provided toclient system 110. In some embodiments, the generic script is a templatefor the script provided to client system 110, and application system 130generates the script based on a generic script based on the context ofhost system 120.

The disclosed embodiments are not limited to tracking users with firstparty cookies. In some embodiments, local storage can be used in placeof first-party cookies to track users. As described herein, tracking(e.g., using local storage or first-party cookies) can enable thevisualization application to provide the user's uploaded images orotherwise provide an experience personalized to the user (or to asession), even when the user opens the visualization application fromdifferent webpages or browsers.

In some embodiments, application system 130 can obtain sessioninformation for a user from a local storage associated with a browser ofclient system 110. In some embodiments, the session information canenable application system 130 to associate an instance of thevisualization application with at least one of the user or a session. Insome instances, such an association can be established without usingcookies to track the user. The session information can enableapplication system 130 to make resources (e.g., uploaded images,products or product selections, or other resources) associated with theat least one of the user or prior session, available to thevisualization application. Accordingly, the user can access theresources, even when engaging with the visualization application ondifferent webpages. For example, a user might start an instance of thevisualization application through a first webpage of a first serviceprovider and upload images of rooms to application system 130. The usermight then exit the visualization application and access a secondwebpage of a second service provider. The user might start anotherinstance of the visualization application through the second webpage.The second instance may have access to the session information on thelocal storage. The second instance may communicate the sessioninformation to application system 130, enabling the user to visualizeproducts in the previously uploaded images of the rooms, without havingto re-upload the rooms.

In some embodiments, a notification can be forwarded by thevisualization application to another system in response to detection ofan event. The notification can be a network request or a JavaScriptcall. The other system can be an analytics system, which can beconnected to the one or more of client system 110, host system 120, orapplication system 130. Such an analytics system can be configured togather, collect, process, or analyze information or data relating tooperations of a computer or network system. In some instances, theanalytics system can be configured to track indicia of user engagement,such as page views, clicks, interactions with the visualizationapplication (e.g., uploading images, positioning products in an image,manipulating the position or orientation of such products), durations ofinteractions, or the like. The analytics system can be configured totrack such indicia for a single user, aggregate such indicia acrossmultiple users, or perform statistical or machine learning analysis ofsuch aggregated indicia.

In some embodiments, an image uploaded by a user can be adjusted fordisplay. Such adjustments can include changing a resolution, size,shape, brightness, or other image characteristics. The adjustments candepend on a device that will be displaying the image. The adjustmentscan depend on the present values of the resolution, size, shape,brightness, or other image characteristics. The adjusted image canoverwrite the uploaded image, or the original image can be retained. Insome instances, when client system 110 includes multiple devices, theuploaded image can be adjusted to appear differently on differentdevices of client system 110. For example, when client system 110includes a desk-top PC and a mobile device, the uploaded image can beadjusted depending on which device it will be displayed on. Suchadjustments can include reducing image size/resolution when beingdisplayed on the mobile device in order to reduce upload/download time.In another example, adjustments can include enhancing image detailsusing a neural network for viewing on a larger monitor of a desk-top PC.In some embodiments, application system 130 can detect a deviceparameter of client system 110 (e.g., display screen size, resolution,network speed, processing speed, memory availability, etc.). Based onthe detected parameter, application system 130 can generate a modifiedversion of the uploaded image for display on client system 110.

In some embodiments, the visualization application can be configured todisplay a product catalog associated with host system 120. The contentsof the catalog can be selected by a vendor associated with host system120 from options provided by a service provider associated withapplication system 130. For example, a service provider system (whichmay be application system 130 or another system) can provide a set ofpotential products (or product categories) to a vendor system (which maybe host system 120 or another system). The potential products can beprovided as a webpage, catalog, menu, or like. For example, the serviceprovider system can host a webpage accessible to the vendor using thevendor system. In some embodiments, information for visualizing thesepotential products (e.g., a three dimensional model, or other suchinformation as described herein) may have been received by the serviceprovider from a manufacturer (or distributor, or the like) of theproduct. In other embodiments, the service provider may have generatedsuch visualization information (e.g., by scanning instances of theproducts, or the like).

Consistent with disclosed embodiments, the service provider system canreceive a selection of a subset of the potential products from thevendor system (e.g., as form data submitted by the vendor system, or thelike). The disclosed embodiments are not limited to any particularmethod of providing selections to the service provider system.

Consistent with disclosed embodiments, when a user selects a control tostart the visualization application on a webpage hosted by host system120, application system 130 can be configured to cause the visualizationapplication to display one or more of the selected potential products.In some embodiments, the webpage can be modified to display thepotential products in the visualization application as a catalog. Inthis manner, a catalog including the visualization functions describedherein can be provided for a vendor without requiring extensiveinvolvement or technical knowledge on the part of the vendor. Forexample, the owner of a local furniture store could interact with adistributor to select products for stocking in the furniture store. Thedistributor could provide a script enabling a webpage of the furniturestore to display a control that starts a visualization application. Thevisualization application could display a catalog showing the selectedproducts and enabling users of the webpage to display these products inimages as described herein. Such functionality might otherwise be wellbeyond the capabilities of such a store.

In some embodiments, host system 120 can include an interactive kioskmachine. The interactive kiosk can allow for a QR code to be scanned toupload an image. In some instances, such an interactive kiosk machinemay be located in a store, a mall, office building, or other publicspace. In some embodiments, the location or the machine can beassociated with a retailer or a store.

In some embodiments, surfaces depicted in the uploaded image or themodified image can include landscaping, house exterior, roof, door,garage door, window, side paneling, or the like.

FIG. 7 depicts a method for interacting with a visualizationapplication, in accordance with disclosed embodiments. Such interactionscan occur after modification of the webpage to display the visualizationapplication, such as step 270 as described previously with reference toFIG. 2. In some embodiments, the visualization experience described inthe present disclosure can be generalized to support multiple producttypes. For example, the script can determine the product types (such asflooring, furniture, or the like) based on the website content of hostsystem 120, and load the visualization experience corresponding to thedetermined product type. Furthermore, the visualization application candetermine the desired product type based on contents of an uploadedimage. For example, if only a wall is visible in the uploaded image,then only product or product type that can be applied to the wall, butnot to the floor may be determined to be desirable.

In step 710, a user can interact with client system 110 to select animage. The image can be of a location (indoor or outdoor) where theproduct can be placed. As a non-limiting example, the image can be of aroom. In some embodiments, the graphical user interface can provideoptions for uploading an image as described previously.

In step 720, client system 110 can provide a request to display theselected product in the selected image to application system 130. Therequest can be provided using network 140. The disclosed embodiments arenot limited to any particular method or protocol for providing therequest. In some embodiments, when the image is uploaded by clientsystem 110, the request can include the image. The request can be sentto application system 130 as previously described.

In step 730, application system 130 can identify one or more locationson the uploaded imaged received in the request of step 720. In someembodiments, application system 130 can be configured to identify one ormore locations by performing semantic segmentation of the image. Asdescribed previously, semantic segmentation can be used to identifyimage pixels associated with a shared object represented in the image,thereby enabling objects represented in the image to be identified. As anon-limiting example, semantic segmentation can be used to identifyfloors, walls, countertops, items of furniture (e.g., tables, bookcases,shelving), or the like in the image. Each of these identified portionsof the image can be a location. As described previously, applicationsystem 130 can use one or more machine learning models to perform thesemantic segmentation of the image, such as using a convolutional neuralnetwork built on an AlexNet, VGG-16, GoogLeNet, ResNet, or othersuitable architecture. As described previously, application system 130can be configured to use multiple machine learning models to performsemantic segmentation of the image. The multiple machine learning modelscan be trained to identify differing portions of the image.

In step 740, client system 110 can receive the locations identified instep 730. Client system 110 can be configured to display the uploadedimage with graphical indicators identifying the locations in the image.By way of example, FIG. 8 illustrates an exemplary image with identifiedlocations for portions of the image indicated by graphical indicators,in accordance with disclosed embodiments. Image 800 can be an imageuploaded by client system 110, depicting a room including variousfurniture and decorations. Image 800 can be displayed to the user onclient system 110, including graphical indicators indicating locationsin image 800. As depicted, graphical indicator 802 indicates a locationthat is “Countertops,” graphical indicator 804 indicates a location thatis “Walls,” graphical indicator 806 indicates a location that is“Cabinets,” and graphical indicator 808 indicates a location that is“Floors.” Consistent with disclosed embodiments, the locations may havebeen identified in step 730.

In step 750, client system 110 can receive user selection of locations.In some embodiments, the user selections can be received via userinteractions with the displayed image. In some instances, the user caninteract with the displayed graphical indicators. In such instances,each graphical indicator can be (or trigger display of) an interactiveinterface capable of receiving a user selection. For example, if clientsystem 110 is a mobile device having a touch screen, a user can touchthe one or more of the graphical indicators to select a location on thetouch screen. The disclosed embodiments are not limited to anyparticular type of interactive interface. Exemplary interfaces couldinclude radio buttons, check boxes, a drop-down menu associated with theoverall image (the graphical indicators being menu entries within thedrop-down menu), or the like.

In step 760, application system 130 can receive location selections fromclient system 110. For example, when the user selects graphicalindicator 808 on client system 110, this selection can be communicatedby client system 110 to application system 130.

In step 770, application system 130 generates instructions. In someembodiments, applications system 130 generates instructions to displaythe product in the uploaded image (e.g., image 800) at the selectedlocation. The image of product may be rendered in manner describedpreviously with reference to FIG. 3.

In some embodiments, application system 130 can identify a product typecorresponding to the selected location. For example, when user selectsgraphical indicator 808, application system 130 generates instructionsfor displaying products that are relevant to “Floors,” such as tiles,carpets, etc. to the location corresponding to graphical indicator 808.In some embodiments, the particular product to be displayed can be aproduct deemed relevant to the user of client system 110. For example,if the user has browsed website of host system 120, the last browseditem having a product type corresponding to the selected location (e.g.,carpet when floor is selected) can be the product displayed. In anotherexample, if the user has saved, tagged, or otherwise indicated aninterest in a particular product, that particular product can beselected to be displayed on the selected location if the product typematches the selected location.

Optionally, in some embodiments, application system 130 can, afteridentifying locations in step 730, generate instructions to displayproducts at the identified locations. The products displayed can beproducts that match the product type of the identified locations. Forexample, a product having the type “floor” can be displayed over thefloor segment of the image, while a product having the type “counter”can be displayed over the counter segment of the image.

Optionally, in some embodiments, after receiving location selections instep 760, application system 130 can provide instructions to clientsystem 110 to indicate products available for display at the one or moreselected locations. For example, the user can select “floor” and receiveinstructions to display a menu of flooring options. The disclosedembodiments are not limited to any particular way of displaying suchavailable products. They may be displayed within the interactiveapplication, in other window, or the like. The user can interact withclient system 110 to specify a particular product(s) for display at theone of more selected locations. Application system 130 can receive anindication of such user interactions and generate instructions todisplay the indicated product(s) at the one or more selected locations.

In step 780, client system 110 receives instructions to display productsat the selection locations. In step 790, the visualization applicationdisplays the products on the selected locations according to thereceived instructions on client system 110.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to precise formsor embodiments disclosed. Modifications and adaptations of theembodiments will be apparent from consideration of the specification andpractice of the disclosed embodiments. For example, the describedimplementations include hardware, but systems and methods consistentwith the present disclosure can be implemented with hardware andsoftware. In addition, while certain components have been described asbeing coupled to one another, such components may be integrated with oneanother or distributed in any suitable fashion.

Moreover, while illustrative embodiments have been described herein, thescope includes any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations or alterations based on the presentdisclosure. The elements in the claims are to be interpreted broadlybased on the language employed in the claims and not limited to examplesdescribed in the present specification or during the prosecution of theapplication, which examples are to be construed as nonexclusive.Further, the steps of the disclosed methods can be modified in anymanner, including reordering steps or inserting or deleting steps.

The features and advantages of the disclosure are apparent from thedetailed specification, and thus, it is intended that the appendedclaims cover all systems and methods falling within the true spirit andscope of the disclosure. As used herein, the indefinite articles “a” and“an” mean “one or more.” Similarly, the use of a plural term does notnecessarily denote a plurality unless it is unambiguous in the givencontext. Words such as “and” or “or” mean “and/or” unless specificallydirected otherwise. Further, since numerous modifications and variationswill readily occur from studying the present disclosure, it is notdesired to limit the disclosure to the exact construction and operationillustrated and described, and accordingly, all suitable modificationsand equivalents may be resorted to, falling within the scope of thedisclosure.

As used herein, unless specifically stated otherwise, the term “or”encompasses all possible combinations, except where infeasible. Forexample, if it is stated that a component may include A or B, then,unless specifically stated otherwise or infeasible, the component mayinclude A, or B, or A and B. As a second example, if it is stated that acomponent may include A, B, or C, then, unless specifically statedotherwise or infeasible, the component may include A, or B, or C, or Aand B, or A and C, or B and C, or A and B and C.

The embodiments may further be described using the following clauses:

1. A method for visualizing products in a single-page application,comprising: receiving, by an application system from a client systemduring rendering of a webpage received by the client system from a hostsystem, a request for a script associated with the host system, thescript executable by the client system to perform operations comprising:modifying the webpage to include a visualization application in responseto a selection of a visualization control in the webpage by a user ofthe client system; receiving, by the application system from thevisualization application, a request to display an image of a product ina first image; identifying a location in the first image; and providinginstructions to the visualization application for displaying the imageof the product at the location.

2. The method of clause 1, wherein the visualization application runsindependently of the webpage, enabling the image of the product to bedisplayed without requiring reloading of the webpage.

3. The method of clause 1 or 2, wherein the visualization application ishosted on the application system, and modifying the webpage to includethe visualization application comprises: integrating the visualizationapplication hosted on the application system, into the webpage receivedby the client system from the host system.

4. The method of any preceding clause, wherein the script is executableby the client system to perform additional operations comprising:generating an identifier from information extracted from the webpage;querying the application system for product availability informationusing the identifier; and modifying the webpage to include thevisualization control in response to receiving the product availabilityinformation from the application system.

5. The method of any preceding clause, wherein the method furthercomprises: receiving, by the application system after rendering of thewebpage, a query for the product availability information; andproviding, by the application system in response to the query, theproduct availability information.

6. The method of any preceding clause, wherein identifying the locationin the first image comprises: performing semantic segmentation of thefirst image to identify features in the first image; and selecting thelocation from at least one of the identified features.

7. The method of any preceding clause, wherein identifying the locationin the first image comprises: using a machine learning model to identifythe location in the first image.

8. The method of clause 7, wherein identifying the location in the firstimage using the machine learning model comprises: providing the firstimage to a convolutional neural network selected based on at least oneof a type of the product, the webpage, or the host system.

9. The method of clause 7 or 8, wherein identifying the location in thefirst image using the machine learning model further comprises:determining a perspective and scale of the first image.

10 . The method of any preceding clause, wherein identifying thelocation in the first image further comprises: classifying pixels in thefirst image as corresponding to a surface depicted in the first image.

11. The method of clause 10, wherein providing the instructions to thevisualization application further comprises: providing instructions fordisplaying the image of the product at the surface depicted in the firstimage.

12. The method of clause 10 or 11, wherein the surface is a wall,counter, or floor of a room depicted in the first image.

13. The method of any preceding clause, wherein the instructionsprovided to the visualization application comprise: instructions forgenerating a second image by painting the image of the product at thelocation in the first image.

14. The method of any one of clauses 1 to 12, wherein the method furthercomprises: generating a second image by painting the image of theproduct at the location in the first image; and the instructionsprovided to the visualization application comprise the second image.

15. The method of clause 13 or 14, wherein painting the image of theproduct comprises: classifying pixels in the first image ascorresponding to a surface depicted in the first image; and replacingthe classified pixels in the first image with corresponding pixels ofthe image of the product.

16. The method of any preceding clause, wherein the method furthercomprises: receiving the first image from the client system.

17. The method of any preceding clause, wherein the method furthercomprises: sending, responsive to receipt of the request for the scriptassociated with the host system, the script to the client system.

18. The method of clause 17, wherein the method further comprises:executing the script, by the client system, to establish a session forthe user using a first-party cookie; and obtaining, by the applicationsystem, the first image using information associated with the session.

19. The method of any preceding clause, wherein the instructions includea three-dimensional model image of the product and metadata specifyingthe scaling and placement of the three-dimensional model image in thefirst image.

20. The method of any preceding clause, further comprising: identifying,based on the request, a context of the webpage; and generating thescript based on the context provided in the request.

21. The method of any preceding clause, wherein the script is generatedfrom a template associated with the context provided in the request.

22. The method of any preceding clause, further comprising: obtaining,by the application system from local storage associated with a browserof the client system, session information for a user; and obtaining, bythe application system, the first image using the session information.

23. The method of any preceding clause, wherein, the method furthercomprises: detecting a device parameter of the client system; andgenerating a modified first image based on the detected deviceparameter; and the instructions comprise instructions for displaying theimage of the product at the location in the modified first image.

24. The method of any preceding clause, further comprising: providing aset of potential products to a vendor system including the product; andreceiving a selection of a subset of the potential products from thevendor system for display on the webpage by the visualizationapplication.

25. The method of any preceding clause, wherein: the host system is aninteractive kiosk machine; and a QR code can be scanned to upload imagesonto the interactive kiosk machine.

26. The method of any one of clauses 10, 11, 12, or 15, wherein thesurface is landscaping, house exterior, roof, door, garage door, window,or side paneling

27. A system for visualizing products in a single-page application,comprising: at least one processor; and at least one memory containingfirst instructions that, when executed by the at least one processor,cause the system to perform operations comprising: receiving a webpagefrom a host system; modifying the webpage to include a visualizationapplication in response to a selection of a visualization control in thewebpage by a user of the system; providing, to an application system, inresponse to a user interaction with the visualization application, arequest to display an image of a product in a first image; andreceiving, from the application system, second instructions fordisplaying the image of the product in the first image.

28. The system of clause 27, wherein the operations further comprise:extracting information from the webpage, the extracted informationincluding product information or vendor information; generating anidentifier using the extracted information; and querying the applicationsystem for product availability information using the identifier.

29. The system of clause 28, wherein the operations further comprise:modifying the webpage to include the visualization control in responseto receiving the product availability information from the applicationsystem.

30. The system of clause 29, wherein a type of the visualization controldepends on a type of the product availability information.

31. The system of clause 29 or 30, wherein the webpage displays multipleproducts, and modifying the webpage to include the visualization controlcomprises: modifying the webpage to include visualization controls forat least some of the multiple products.

32. The system of any one of clauses 29 to 31, wherein modifying thewebpage to include the visualization control comprises: unhiding thevisualization control or adding the visualization control to a documentobject model of the webpage.

33. The system of any one of clauses 27 to 32, wherein the operationsfurther comprise: providing the first image to the application system orproviding an indication of the first image to the application system.

34. The system of any one of clauses 27 to 33, wherein the host systemis associated with a retailer and the application system is associatedwith a visualization services provider.

35. The system of any one of clauses 27 to 34, wherein receiving thesecond instructions for displaying the image of the product in the firstimage, comprises: receiving a second image generated by painting theimage of the product into the first image.

36. The system of clause 35, wherein the image of the product is paintedinto the first image at a location identified by a machine learningmodel.

37. The system of any one of clauses 27 to 36, wherein receiving thesecond instructions for displaying the image of the product in the firstimage, comprises: receiving a three-dimensional model of the image ofthe product and metadata specifying the scaling and placement of thethree-dimensional model at a location in the first image.

38. The system of clause 37, wherein the operations further comprise:updating, in response to a user interaction with the visualizationapplication, a position or orientation of the displayed image of theproduct in the first image using the three-dimensional model.

39. The system of any one of clauses 27 to 38, wherein: the product israndomly selected by the application system based on productavailability information.

40. The system of any one of clauses 27 to 38, wherein: the productcomprises a predetermined product.

41. The system of any one of clauses 27 to 38, wherein: thevisualization control is associated with the product.

42. The system of clause 41, wherein: the visualization controlcomprises a selectable advertisement for the product.

43. The system of any one of clauses 27 to 42, further comprisingforwarding, by the visualization application to an analytics system, anotification in response to detection of an event.

44. A system for enabling visualization of products in a single-pageapplication, comprising: at least one processor; and at least one memorycontaining first instructions that, when executed by the at least oneprocessor, cause the system to perform operations comprising: receiving,from a client system in response to a user interaction with avisualization application in a webpage running on the client system, arequest to display an image of a product in a first image; andproviding, to the client system, second instructions for displaying theimage of the product in the first image.

45. The system of clause 44, wherein providing the second instructionsfor displaying the image of the product in the first image, comprises:identifying a location in the first image, and providing instructions tothe visualization application for displaying the image of the product atthe location.

46. The system of clause 45, wherein identifying the location in thefirst image, comprises: performing semantic segmentation of the firstimage to identify features in the first image; and selecting thelocation from at least one of the identified features.

47. The system of any one of clauses 44 to 45, wherein identifying thelocation in the first image, comprises: using machine learning toidentify the location in the first image.

48. The system of clause 47, wherein using machine learning to identifythe location in the first image, comprises: providing the first image toa convolutional neural network selected based on at least one of a typeof the product, the webpage, or the host system.

49. The system of any one of clauses 47 to 48, wherein using machinelearning to identify the location in the first image, further comprises:determining a perspective and scale of the first image.

50. The system of any one of clauses 45 to 49, wherein identifying thelocation in the first image, further comprises: classifying pixels inthe first image as corresponding to a surface depicted in the firstimage.

51. The system of clause 50, wherein providing second instructions fordisplaying the image of the product in the first image, furthercomprises: providing instructions for displaying the image of theproduct at the surface depicted in the first image.

52. The system of clause 50 or 51, wherein the surface is a wall,counter, or a floor of a room depicted in the first image, and providingthe instructions for displaying the image of the product at the surfacedepicted in the first image comprises: providing instructions fordisplaying the image of the product at any one of the wall, counter, orfloor of the room depicted in the first image.

53. The system of any one of clauses 44 to 52, wherein the secondinstructions provided to the client system comprise: instructions forgenerating a second image by painting the image of the product at thelocation in the first image.

54. The system of any one of clauses 44 to 53, wherein the operationsfurther comprise: generating a second image by painting the image of theproduct at the location in the first image; and the second instructionsprovided to the client system comprise the second image.

55. The system of any one of clauses 53 to 54, wherein painting theimage of the product comprises: classifying pixels in the first image ascorresponding to a surface depicted in the first image; and replacingthe classified pixels in the first image with corresponding pixels ofthe image of the product.

56. A method for visualizing products in a single-page application,comprising: receiving, by an application system from a client systemduring rendering of a webpage received by the client system from a hostsystem, a first request for a script associated with the host system,the script executable by the client system to perform operationscomprising: modifying the webpage to include a visualization applicationin response to a selection of a visualization control in the webpage bya user of the client system; receiving, by the application system fromthe visualization application, a second request to display an augmentedreality image; providing, by the application system to the clientsystem, instructions to cause the display, by the client system andusing a native tool of the client system, of the augmented reality imagein image data of an environment, the augmented reality image depicting aproduct at a location in the environment.

57. The method of claim 56, wherein: the product is randomly selected bythe application system based on product availability information.

58. The method of claim 56, wherein: the product comprises apredetermined product.

59. The method of any one of claims 56 to 58, wherein: the visualizationcontrol is associated with the product.

60. The method of any one of claims 56 to 59, further comprising:identify, based on the first request, a context of the webpage; andgenerating the script based on the context provided in the firstrequest.

61. The method of any one of claims 56 to 59, wherein the script isgenerated from a template associated with a context provided in thefirst request.

62. The system of any one of claims 56 to 61, further comprisingforwarding, by the visualization application to an analytics system, anotification in response to detection of an event.

63. The method of any one of claims 56 to 62, wherein the image data ofthe environment comprises a live stream of images acquired by the clientsystem.

64. The method of any one of claims 56 to 63, wherein the location isfixed with respect to the environment.

65. A method for visualizing products in a single-page application,comprising: receiving, by an application system from a client systemduring rendering of a webpage received by the client system from a hostsystem, a request for a script associated with the host system, thescript executable by the client system to perform operations comprising:modifying the webpage to include a visualization application in responseto a selection of a visualization control in the webpage by a user ofthe client system; receiving, by the application system from thevisualization application, a display request; identifying a location ina first image associated with the display request; providing, to theclient system for display in the visualization application, anindication of the location; receiving a selection of the indication; andproviding, based on the selection, instructions to the visualizationapplication for displaying an image of a product associated with thelocation in the location.

66. The method of claim 65, wherein: the operations further compriseidentifying a product type corresponding to the location; and theinstructions are provided based in part on the product typecorresponding to the location.

67. The method of any one of claims 65 to 66, wherein identifying thelocation in the first image comprises: performing semantic segmentationof the first image to identify features in the first image; andidentifying the location from at least one of the identified features.

68. The method of any one of claims 65 to 66, wherein identifying thelocation in the first image associated with the display requestcomprises: using a machine learning model to identify the location inthe first image.

69. The method of any one of claims 65 to 66, wherein identifying thelocation in the first image associated with the display requestcomprises: providing the first image to a convolutional neural networkselected based on at least one of a type of the product, the webpage, orthe host system.

70. The method of any one of claims 65 to 66, wherein identifying thelocation in the first image associated with the display requestcomprises: classifying pixels in the first image as corresponding to asurface depicted in the first image; and identifying the location basedon the classified pixels in the first image.

71. The method of claim 70, wherein the surface is landscaping, houseexterior, roof, door, garage door, window, or side paneling.

72. The method of any one of claims 65 to 71, further comprising:obtaining, by the application system from local storage associated witha browser of the client system, session information for the user; andobtaining, by the application system, the first image using the sessioninformation.

73. The method of any one of claims 65 to 72, wherein: the methodfurther comprises:

detecting a device parameter of the client system; and generating amodified first image based on the detected device parameter; and theinstructions comprise instructions for displaying the image of theproduct at the location in the modified first image.

74. The method of any one of claims 65 to 73, further comprising:providing a set of potential products to a vendor system including theproduct; and receiving a selection of subset from the vendor system fordisplay on the webpage by the visualization application.

75. The method of any one of claims 65 to 74, wherein the host system isan interactive kiosk machine; and a QR code can be scanned to uploadimages onto the interactive kiosk machine.

76. The method of any one of claims 65 to 75, further comprisingforwarding, by the visualization application to an analytics system, anotification in response to detection of an event.

77. A system for visualizing products in a single-page application,comprising: at least one processor; and at least one memory containingfirst instructions that, when executed by the at least one processor,cause the system to perform first operations comprising: receiving, froma client system and during rendering of a webpage received by the clientsystem in response to a first request, a second request for a script,the script associated with a host system and executable by the clientsystem to perform second operations comprising: modifying the webpage toinclude a visualization control; displaying a visualization applicationin response to a selection of the visualization control in the webpageby a user of the client system; transmitting, to an application system,a third request to display a product in a first image; receiving, fromthe application system, a second instructions to display the product atan identified location in the first image; and displaying, by thevisualization application, the product in accordance with the secondinstructions; and providing the script to the client system.

78. The system of clause 77, wherein the webpage depicts multipleproducts; the second operations comprise receiving, through userinteractions with the webpage, a selection of one of the multipleproducts as the product; and the third request is transmitted inresponse to the selection of the product.

79. The system of clause 77, wherein the second operations furthercomprise generating an identifier of the product based on informationstored in the webpage at a predetermined location.

80. The system of any one of clauses 77 to 79, wherein the secondinstructions comprise a third image generated by adding an image of theproduct to the first image at the identified location.

81. The system of any one of clauses 77 to 79, wherein the secondinstructions include a three-dimensional model of the product andmetadata specifying the scaling and placement of the three-dimensionalmodel in the first image.

82. The system of any one of clauses 77 or 79 to 81, wherein the secondrequest and the third request are transmitted in a single request.

83. The system of any one of clauses 77 to 82, wherein the firstoperations comprise receiving the first and second requests by the hostsystem.

84. The system of any one of clauses 77 to 83, wherein the firstoperations comprise providing the script by the application system.

85. The system of any one of clauses 77 to 84, wherein the secondoperations further comprise displaying the visualization application inthe webpage.

86. The system of any one of clauses 77 to 85, wherein the secondoperations further comprise displaying the visualization application ina new tab or a new webpage.

87. A system for visualizing products in a single-page application,comprising: at least one processor; and at least one memory containingfirst instructions that, when executed by the at least one processor,cause the system to perform first operations comprising: receiving afirst request for a script associated with a host system, the scriptexecutable by a client system to perform second operations comprising:modifying a webpage displayed by the client system to include avisualization application in response to a selection of a visualizationcontrol in the webpage by a user of the client system; receiving asecond request to display a product in a first image; identifying alocation in the first image, the location identified using a machinelearning model; and providing second instructions to the visualizationapplication for displaying the product at the location.

88. The system of clause 87, wherein the first operations furthercomprise generating a third image by painting a second image of theproduct into the first image at the location; and the secondinstructions comprise the third image.

89. The system of clause 87, wherein the second instructions comprise athree-dimensional model of the product and metadata specifying thescaling and placement of the three-dimensional model at the location inthe first image.

90. The system of any one of clauses 87 to 89, wherein the firstoperations further comprise performing semantic segmentation of thefirst image using a convolutional neural network.

91. The system of clause 90, wherein the convolutional neural network isbuilt on at least one of AlexNet, VGG-16, googLeNet, or ResNetarchitecture.

92. The system of clause 90, wherein the convolutional neural network istrained to perform at least one of region-based semantic segmentation orfull convolutional network-based semantic segmentation.

93. The system of any one of clauses 87 to 92, wherein the secondoperations further comprise providing an identifier of the product, theidentifier generated based on information stored in the webpage at apredetermined location.

94. The system of any one of clauses 87 to 93, wherein the firstoperations further comprise receiving a query concerning an availabilityof the product.

95. The system of any one of clauses 87 to 94, wherein the query and thefirst request are received in a same step.

96. The system of any one of clauses 87 to 95, wherein the firstoperations further comprise transmitting the script to the host system.

97. A method for visualizing products in a single-page application,comprising: receiving, by an application system from a client systemduring rendering of a webpage received by the client system from a hostsystem, a request for a script associated with the host system , thescript executable by the client system to perform operations comprising:modifying the webpage to include a visualization application in responseto a selection of a visualization control in the webpage by a user ofthe client system; receiving, by the application system from thevisualization application, a request to display a product in a firstimage; identifying a location in the first image using a machinelearning model; and providing instructions to the visualizationapplication for displaying the product at the location.

98. The method of claim 97, wherein the script is executable by theclient system to perform additional operations comprising: generating anidentifier from information extracted from the webpage; querying theapplication system for product availability information using theidentifier; and modifying the webpage to include the visualizationcontrol in response to receiving the product availability informationfrom the application system.

99. The method of claim 98, wherein the method further comprises:receiving, by the application system after rendering of the webpage, aquery for the product availability information; and providing, by theapplication system in response to the query, the product availabilityinformation.

100. The method of claim 97, wherein identifying the location in thefirst image using the machine learning model comprises: providing thefirst image to a convolutional neural network selected based on at leastone of a type of the product, the webpage, or the host system.

101. The method of claim 97, wherein identifying the location in thefirst image using the machine learning model further comprises:determining a perspective and scale of the first image.

102. The method of claim 97, wherein identifying the location in thefirst image further comprises: classifying pixels in the first image ascorresponding to a surface depicted in the first image.

103. The method of claim 102, wherein the surface is a wall, counter, orfloor of a room depicted in the first image.

104. The method of claim 97, wherein: the method further comprisesgenerating a third image by painting a second image of the product intothe first image at the location; and the instructions comprise the thirdimage.

105. The method of claim 104, wherein painting the second image of theproduct comprises: classifying pixels in the first image ascorresponding to a surface depicted in the first image; and replacingthe classified pixels in the first image with corresponding pixels ofthe second image.

106. The method of claim 97, wherein the method further comprises:receiving the first image from the client system.

107. The method of claim 97, wherein: the script is executable by theclient system to establish a session for the user using a first-partycookie; and the application system uses information associated with thesession to obtain the first image.

108. The method of claim 97, wherein the instructions include athree-dimensional model of the product and metadata specifying thescaling and placement of the three-dimensional model in the first image.

109. A system for visualizing products in a single-page application,comprising: at least one processor; and at least one memory containingfirst instructions that, when executed by the at least one processor,cause the system to perform operations comprising: receiving a webpagefrom a host system; modifying the webpage to include a visualizationapplication in response to a selection of a visualization control in thewebpage by a user of the system; providing, to an application system inresponse to user interactions with the visualization application, arequest to display a product in a first image; and receiving, from theapplication system, second instructions for displaying the product.

110. The system of claim 109, wherein the operations further comprise:extracting information from the webpage, the extracted informationincluding product information or vendor information; generating anidentifier using the extracted information; and querying the applicationsystem for product availability information using the identifier.

111. The system of claim 110, wherein the operations further comprise:modifying the webpage to include the visualization control in responseto receiving the product availability information from the applicationsystem.

112. The system of claim 111, wherein a type of the visualizationcontrol depends on a type of the product availability information.

113. The system of claim 111, wherein the webpage displays multipleproducts and modifying the webpage to include the visualization controlincludes modifying the webpage to include visualization controls for atleast some of the multiple products.

114. The system of claim 111, wherein modifying the webpage to includethe visualization control comprises unhiding the visualization controlor adding the visualization control to a document object model of thewebpage.

115. The system of claim 109, wherein the operations further comprise:providing the first image to the application system or providing anindication of the first image to the application system.

116. The system of claim 109, wherein the host system is associated witha retailer and the application system is associated with a visualizationservices provider.

117. The system of claim 109, wherein receiving second instructions todisplay the product comprises: receiving a third image generated bypainting a second image of the product into the first image at alocation identified by a machine learning model.

118. The system of claim 109, wherein receiving second instructions todisplay the product comprises: receiving a three-dimensional model ofthe product and metadata specifying the scaling and placement of thethree-dimensional model at a location in the first image.

119. The system of claim 118, wherein the operations further comprise:updating, in response to user interactions with the visualizationapplication, a position or orientation of the displayed product in thefirst image using the three-dimensional model.

Other embodiments will be apparent from consideration of thespecification and practice of the embodiments disclosed herein. It isintended that the specification and examples be considered as exampleonly, with a true scope and spirit of the disclosed embodiments beingindicated by the following claims.

1. A method for visualizing products in a single-page application,comprising: receiving, by an application system from a client systemduring rendering of a webpage received by the client system from a hostsystem, a first request for a script associated with the host system,the script executable by the client system to perform operationscomprising: modifying the webpage to include a visualization applicationin response to a selection of a visualization control in the webpage bya user of the client system; receiving, by the application system fromthe visualization application, a second request to display an augmentedreality image; and providing, by the application system to the clientsystem, instructions to cause the display, by the client system andusing a native tool of the client system, of the augmented reality imagein image data of an environment, the augmented reality image depicting aproduct at a location in the environment.
 2. The method of claim 1,wherein: the product is randomly selected by the application systembased on product availability information.
 3. The method of claim 1,wherein: the product comprises a predetermined product.
 4. The method ofclaim 1, wherein: the visualization control is associated with theproduct.
 5. The method of claim 1, further comprising: identify, basedon the first request, a context of the webpage; and generating thescript based on the context provided in the first request.
 6. The methodof claim 1, wherein the script is generated from a template associatedwith a context provided in the first request.
 7. The system of claim 1,further comprising: forwarding, by the visualization application to ananalytics system, a notification in response to detection of an event.8. The method of claim 1, wherein the image data of the environmentcomprises a live stream of images acquired by the client system.
 9. Themethod of claim 1, wherein the location is fixed with respect to theenvironment.
 10. A method for visualizing products in a single-pageapplication, comprising: receiving, by an application system from aclient system during rendering of a webpage received by the clientsystem from a host system, a request for a script associated with thehost system, the script executable by the client system to performoperations comprising: modifying the webpage to include a visualizationapplication in response to a selection of a visualization control in thewebpage by a user of the client system; receiving, by the applicationsystem from the visualization application, a display request;identifying a location in a first image associated with the displayrequest; providing, to the client system for display in thevisualization application, an indication of the location; receiving aselection of the indication; and providing, based on the selection,instructions to the visualization application for displaying an image ofa product associated with the location in the location.
 11. The methodof claim 10, wherein: the operations further comprise identifying aproduct type corresponding to the location; and the instructions areprovided based in part on the product type corresponding to thelocation.
 12. The method of claim 10, wherein identifying the locationin the first image comprises: performing semantic segmentation of thefirst image to identify features in the first image; and identifying thelocation from at least one of the identified features.
 13. The method ofclaim 10, wherein identifying the location in the first image associatedwith the display request comprises: using a machine learning model toidentify the location in the first image.
 14. The method of claim 10,wherein identifying the location in the first image associated with thedisplay request comprises: providing the first image to a convolutionalneural network selected based on at least one of a type of the product,the webpage, or the host system.
 15. The method of claim 10, whereinidentifying the location in the first image associated with the displayrequest comprises: classifying pixels in the first image ascorresponding to a surface depicted in the first image; and identifyingthe location based on the classified pixels in the first image.
 16. Themethod of claim 15, wherein the surface is landscaping, house exterior,roof, door, garage door, window, or side paneling.
 17. The method ofclaim 10, further comprising: obtaining, by the application system fromlocal storage associated with a browser of the client system, sessioninformation for the user; and obtaining, by the application system, thefirst image using the session information.
 18. The method of claim 10,wherein: the method further comprises: detecting a device parameter ofthe client system; and generating a modified first image based on thedetected device parameter; and the instructions comprise instructionsfor displaying the image of the product at the location in the modifiedfirst image.
 19. The method of claim 10, further comprising: providing aset of potential products to a vendor system including the product; andreceiving a selection of subset from the vendor system for display onthe webpage by the visualization application.
 20. The method of claim10, wherein: the host system comprises an interactive kiosk machineincluding a QR code scanner; and the operations further comprise:scanning an QR code using the QR code scanner; and uploading, based onthe scanned QR code, the first image.
 21. The method of claim 10,further comprising: forwarding, by the visualization application to ananalytics system, a notification in response to detection of an event.